-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
@uppy/aws-s3-multipart: Fix escaping issue with client signed request #5006
@uppy/aws-s3-multipart: Fix escaping issue with client signed request #5006
Conversation
Diff output filesdiff --git a/packages/@uppy/aws-s3-multipart/lib/createSignedURL.js b/packages/@uppy/aws-s3-multipart/lib/createSignedURL.js
index 7ce2b7c..c6d54e0 100644
--- a/packages/@uppy/aws-s3-multipart/lib/createSignedURL.js
+++ b/packages/@uppy/aws-s3-multipart/lib/createSignedURL.js
@@ -48,6 +48,9 @@ async function hash(key, data) {
} = globalThis.crypto;
return subtle.sign(algorithm, await generateHmacKey(key), ec.encode(data));
}
+function percentEncode(c) {
+ return `%${c.charCodeAt(0).toString(16).toUpperCase()}`;
+}
export default async function createSignedURL(_ref2) {
let {
accountKey,
@@ -62,7 +65,7 @@ export default async function createSignedURL(_ref2) {
} = _ref2;
const Service = "s3";
const host = `${bucketName}.${Service}.${Region}.amazonaws.com`;
- const CanonicalUri = `/${encodeURI(Key)}`;
+ const CanonicalUri = `/${encodeURI(Key).replace(/[;?:@&=+$,#!'()*]/g, percentEncode)}`;
const payload = "UNSIGNED-PAYLOAD";
const requestDateTime = new Date().toISOString().replace(/[-:]|\.\d+/g, "");
const date = requestDateTime.slice(0, 8); |
Needs e2e test? |
Thanks a lot for looking into this and writing a test. I'm just wondering why this is reverse engineered, can't we use the AWS SDK for this to make sure we stay aligned? |
The AWS SDK is enormous, the goal was to limit the grow of the bundle size. |
…ars in query on self-signing request. Thanks to review from @aduh95
6b5e128
to
3ab6024
Compare
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the work 🙌
* main: (90 commits) crash if trying to set path to / (#5003) fix `super.toggleCheckbox` bug (#5004) @uppy/aws-s3-multipart: fix escaping issue with client signed request (#5006) add missing exports (#5009) @uppy/transloadit: migrate to TS (#4987) @uppy/utils: fix `RateLimitedQueue#wrapPromiseFunction` types (#5007) @uppy/golden-retriever: migrate to TS (#4989) Bump follow-redirects from 1.15.4 to 1.15.6 (#5002) meta: fix `resize-observer-polyfill` types (#4994) @uppy/core: various type fixes (#4995) @uppy/utils: fix `findAllDOMElements` type (#4997) @uppy/status-bar: fix `recoveredState` type (#4996) @uppy/utils: fix `AbortablePromise` type (#4988) Fix breadcrumbs (#4986) @uppy/drag-drop: refactor to TypeScript (#4983) @uppy/webcam: refactor to TypeScript (#4870) @uppy/url: migrate to TS (#4980) @uppy/zoom: refactor to TypeScript (#4979) @uppy/unsplash: refactor to TypeScript (#4979) @uppy/onedrive: refactor to TypeScript (#4979) ...
| Package | Version | Package | Version | | ------------------------- | ------- | ------------------------- | ------- | | @uppy/audio | 1.1.8 | @uppy/progress-bar | 3.1.1 | | @uppy/aws-s3-multipart | 3.11.0 | @uppy/provider-views | 3.11.0 | | @uppy/box | 2.3.0 | @uppy/react | 3.3.0 | | @uppy/companion | 4.13.0 | @uppy/remote-sources | 1.2.0 | | @uppy/companion-client | 3.8.0 | @uppy/screen-capture | 3.2.0 | | @uppy/compressor | 1.1.2 | @uppy/status-bar | 3.3.1 | | @uppy/core | 3.10.0 | @uppy/thumbnail-generator | 3.1.0 | | @uppy/dashboard | 3.8.0 | @uppy/transloadit | 3.6.0 | | @uppy/drag-drop | 3.1.0 | @uppy/tus | 3.5.4 | | @uppy/drop-target | 2.0.5 | @uppy/unsplash | 3.3.0 | | @uppy/dropbox | 3.3.0 | @uppy/url | 3.6.0 | | @uppy/facebook | 3.3.0 | @uppy/utils | 5.7.5 | | @uppy/golden-retriever | 3.2.0 | @uppy/webcam | 3.4.0 | | @uppy/google-drive | 3.5.0 | @uppy/zoom | 2.3.0 | | @uppy/instagram | 3.3.0 | uppy | 3.24.0 | | @uppy/onedrive | 3.3.0 | | | - @uppy/box,@uppy/companion-client,@uppy/provider-views,@uppy/status-bar: fix type imports (Antoine du Hamel / #5038) - @uppy/aws-s3-multipart: mark `opts` as optional (Antoine du Hamel / #5039) - e2e: bump Cypress version (Antoine du Hamel / #5034) - @uppy/react: refactor to TS (Antoine du Hamel / #5012) - @uppy/core: refine type of private variables (Antoine du Hamel / #5028) - @uppy/dashboard: refine type of private variables (Antoine du Hamel / #5027) - @uppy/drag-drop: refine type of private variables (Antoine du Hamel / #5026) - @uppy/status-bar: refine type of private variables (Antoine du Hamel / #5025) - @uppy/remote-sources: migrate to TS (Merlijn Vos / #5020) - @uppy/dashboard: refine option types (Antoine du Hamel / #5022) - @uppy/dashboard: add new `autoOpen` option (Chris Grigg / #5001) - @uppy/core: fix some type errors (Antoine du Hamel / #5015) - @uppy/audio,@uppy/dashboard,@uppy/drop-target,@uppy/webcam: add missing exports (Antoine du Hamel / #5014) - meta: Bump webpack-dev-middleware from 5.3.3 to 5.3.4 (dependabot[bot] / #5013) - @uppy/dashboard: refactor to TypeScript (Antoine du Hamel / #4984) - @uppy/companion: improve error msg (Mikael Finstad / #5010) - @uppy/aws-s3-multipart: refactor to TS (Antoine du Hamel / #4902) - @uppy/dashboard: refactor to stable lifecycle method (Antoine du Hamel / #4999) - @uppy/companion: crash if trying to set path to / (Mikael Finstad / #5003) - @uppy/provider-views: fix `super.toggleCheckbox` bug (Mikael Finstad / #5004) - @uppy/aws-s3-multipart: fix escaping issue with client signed request (Hiroki Shimizu / #5006) - @uppy/drag-drop,@uppy/progress-bar: add missing exports (Antoine du Hamel / #5009) - @uppy/transloadit: migrate to TS (Merlijn Vos / #4987) - @uppy/utils: fix `RateLimitedQueue#wrapPromiseFunction` types (Antoine du Hamel / #5007) - @uppy/golden-retriever: migrate to TS (Merlijn Vos / #4989) - meta: Bump follow-redirects from 1.15.4 to 1.15.6 (dependabot[bot] / #5002) - meta: fix `resize-observer-polyfill` types (Antoine du Hamel / #4994) - @uppy/core: various type fixes (Antoine du Hamel / #4995) - @uppy/utils: fix `findAllDOMElements` type (Antoine du Hamel / #4997) - @uppy/status-bar: fix `recoveredState` type (Antoine du Hamel / #4996) - @uppy/utils: fix `AbortablePromise` type (Antoine du Hamel / #4988) - @uppy/core,@uppy/provider-views: Fix breadcrumbs (Evgenia Karunus / #4986) - @uppy/drag-drop: refactor to TypeScript (Antoine du Hamel / #4983) - @uppy/webcam: refactor to TypeScript (Antoine du Hamel / #4870) - @uppy/url: migrate to TS (Merlijn Vos / #4980) - @uppy/zoom: refactor to TypeScript (Murderlon / #4979) - @uppy/unsplash: refactor to TypeScript (Murderlon / #4979) - @uppy/onedrive: refactor to TypeScript (Murderlon / #4979) - @uppy/instagram: refactor to TypeScript (Murderlon / #4979) - @uppy/google-drive: refactor to TypeScript (Murderlon / #4979) - @uppy/facebook: refactor to TypeScript (Murderlon / #4979) - @uppy/dropbox: refactor to TypeScript (Murderlon / #4979) - @uppy/box: refactor to TypeScript (Murderlon / #4979) - @uppy/utils: migrate RateLimitedQueue to TS (Merlijn Vos / #4981) - @uppy/thumbnail-generator: migrate to TS (Merlijn Vos / #4978) - @uppy/screen-capture: migrate to TS (Merlijn Vos / #4965) - @uppy/companion-client: Replace Provider.initPlugin with composition (Merlijn Vos / #4977)
| Package | Version | Package | Version | | ------------------------- | ------------ | ------------------------- | ------------ | | @uppy/angular | 0.7.0-beta.1 | @uppy/progress-bar | 4.0.0-beta.1 | | @uppy/audio | 2.0.0-beta.1 | @uppy/provider-views | 4.0.0-beta.1 | | @uppy/aws-s3 | 4.0.0-beta.1 | @uppy/react | 4.0.0-beta.1 | | @uppy/aws-s3-multipart | 4.0.0-beta.1 | @uppy/redux-dev-tools | 4.0.0-beta.1 | | @uppy/box | 3.0.0-beta.1 | @uppy/remote-sources | 2.0.0-beta.1 | | @uppy/companion | 5.0.0-beta.1 | @uppy/screen-capture | 4.0.0-beta.1 | | @uppy/companion-client | 4.0.0-beta.1 | @uppy/status-bar | 4.0.0-beta.1 | | @uppy/compressor | 2.0.0-beta.1 | @uppy/store-default | 4.0.0-beta.1 | | @uppy/core | 4.0.0-beta.1 | @uppy/store-redux | 4.0.0-beta.1 | | @uppy/dashboard | 4.0.0-beta.1 | @uppy/svelte | 4.0.0-beta.1 | | @uppy/drag-drop | 4.0.0-beta.1 | @uppy/thumbnail-generator | 4.0.0-beta.1 | | @uppy/drop-target | 3.0.0-beta.1 | @uppy/transloadit | 4.0.0-beta.1 | | @uppy/dropbox | 4.0.0-beta.1 | @uppy/tus | 4.0.0-beta.1 | | @uppy/facebook | 4.0.0-beta.1 | @uppy/unsplash | 4.0.0-beta.1 | | @uppy/file-input | 4.0.0-beta.1 | @uppy/url | 4.0.0-beta.1 | | @uppy/form | 4.0.0-beta.1 | @uppy/utils | 6.0.0-beta.1 | | @uppy/golden-retriever | 4.0.0-beta.1 | @uppy/vue | 2.0.0-beta.1 | | @uppy/google-drive | 4.0.0-beta.1 | @uppy/webcam | 4.0.0-beta.1 | | @uppy/image-editor | 3.0.0-beta.1 | @uppy/xhr-upload | 4.0.0-beta.1 | | @uppy/informer | 4.0.0-beta.1 | @uppy/zoom | 3.0.0-beta.1 | | @uppy/instagram | 4.0.0-beta.1 | uppy | 4.0.0-beta.1 | | @uppy/onedrive | 4.0.0-beta.1 | | | - @uppy/vue: migrate to Composition API with TS & drop Vue 2 support (Merlijn Vos / #5043) - @uppy/angular: upgrade to Angular 17.x and to TS 5.4 (Antoine du Hamel / #5008) - @uppy/svelte: remove UMD output and make it use newer types (Antoine du Hamel / #5023) - @uppy/companion-client,@uppy/provider-views,@uppy/status-bar: fix type imports (Antoine du Hamel / #5038) - @uppy/aws-s3-multipart: mark `opts` as optional (Antoine du Hamel / #5039) - e2e: bump Cypress version (Antoine du Hamel / #5034) - @uppy/react: remove `prop-types` dependency (Antoine du Hamel / #5031) - @uppy/progress-bar: remove default target (Antoine du Hamel / #4971) - @uppy/status-bar: remove default target (Antoine du Hamel / #4970) - @uppy/react: remove `Wrapper.ts` (Antoine du Hamel / #5032) - @uppy/react: refactor to TS (Antoine du Hamel / #5012) - @uppy/core: refine type of private variables (Antoine du Hamel / #5028) - @uppy/dashboard: refine type of private variables (Antoine du Hamel / #5027) - @uppy/drag-drop: refine type of private variables (Antoine du Hamel / #5026) - @uppy/status-bar: refine type of private variables (Antoine du Hamel / #5025) - @uppy/remote-sources: migrate to TS (Merlijn Vos / #5020) - @uppy/dashboard: refine option types (Antoine du Hamel / #5022) - @uppy/dashboard: add new `autoOpen` option (Chris Grigg / #5001) - @uppy/aws-s3-multipart,@uppy/tus,@uppy/utils,@uppy/xhr-upload: Make `allowedMetaFields` consistent (Merlijn Vos / #5011) - @uppy/core: fix some type errors (Antoine du Hamel / #5015) - @uppy/audio,@uppy/dashboard,@uppy/drop-target,@uppy/webcam: add missing exports (Antoine du Hamel / #5014) - meta: Bump webpack-dev-middleware from 5.3.3 to 5.3.4 (dependabot[bot] / #5013) - @uppy/dashboard: refactor to TypeScript (Antoine du Hamel / #4984) - @uppy/companion: improve error msg (Mikael Finstad / #5010) - @uppy/aws-s3-multipart: refactor to TS (Antoine du Hamel / #4902) - @uppy/dashboard: refactor to stable lifecycle method (Antoine du Hamel / #4999) - @uppy/companion: crash if trying to set path to / (Mikael Finstad / #5003) - @uppy/provider-views: fix `super.toggleCheckbox` bug (Mikael Finstad / #5004) - @uppy/aws-s3-multipart: fix escaping issue with client signed request (Hiroki Shimizu / #5006) - @uppy/drag-drop,@uppy/progress-bar: add missing exports (Antoine du Hamel / #5009) - @uppy/transloadit: migrate to TS (Merlijn Vos / #4987) - @uppy/utils: fix `RateLimitedQueue#wrapPromiseFunction` types (Antoine du Hamel / #5007) - @uppy/golden-retriever: migrate to TS (Merlijn Vos / #4989) - meta: Bump follow-redirects from 1.15.4 to 1.15.6 (dependabot[bot] / #5002) - meta: fix `resize-observer-polyfill` types (Antoine du Hamel / #4994) - @uppy/core: various type fixes (Antoine du Hamel / #4995) - @uppy/utils: fix `findAllDOMElements` type (Antoine du Hamel / #4997) - @uppy/status-bar: fix `recoveredState` type (Antoine du Hamel / #4996) - @uppy/utils: fix `AbortablePromise` type (Antoine du Hamel / #4988) - @uppy/core,@uppy/provider-views: Fix breadcrumbs (Evgenia Karunus / #4986) - @uppy/drag-drop: refactor to TypeScript (Antoine du Hamel / #4983) - @uppy/webcam: refactor to TypeScript (Antoine du Hamel / #4870) - @uppy/url: migrate to TS (Merlijn Vos / #4980) - @uppy/zoom: refactor to TypeScript (Murderlon / #4979) - @uppy/unsplash: refactor to TypeScript (Murderlon / #4979) - @uppy/onedrive: refactor to TypeScript (Murderlon / #4979) - @uppy/instagram: refactor to TypeScript (Murderlon / #4979) - @uppy/google-drive: refactor to TypeScript (Murderlon / #4979) - @uppy/facebook: refactor to TypeScript (Murderlon / #4979) - @uppy/dropbox: refactor to TypeScript (Murderlon / #4979) - @uppy/box: refactor to TypeScript (Murderlon / #4979) - @uppy/utils: migrate RateLimitedQueue to TS (Merlijn Vos / #4981) - @uppy/thumbnail-generator: migrate to TS (Merlijn Vos / #4978) - @uppy/screen-capture: migrate to TS (Merlijn Vos / #4965) - @uppy/companion-client: Replace Provider.initPlugin with composition (Merlijn Vos / #4977) - uppy: remove legacy bundle (Antoine du Hamel) - meta: include types in npm archive (Antoine du Hamel) - @uppy/angular: fix build (Antoine du Hamel) - meta: Remove generate types from locale-pack (Murderlon) - meta: enable CI on `4.x` branch (Antoine du Hamel) - @uppy/vue: [v4.x] remove manual types (Antoine du Hamel / #4803) - meta: prepare release workflow for beta versions (Antoine du Hamel) | Package | Version | Package | Version | | ------------------------- | ------- | ------------------------- | ------- | | @uppy/audio | 1.1.8 | @uppy/progress-bar | 3.1.1 | | @uppy/aws-s3-multipart | 3.11.0 | @uppy/provider-views | 3.11.0 | | @uppy/box | 2.3.0 | @uppy/react | 3.3.0 | | @uppy/companion | 4.13.0 | @uppy/remote-sources | 1.2.0 | | @uppy/companion-client | 3.8.0 | @uppy/screen-capture | 3.2.0 | | @uppy/compressor | 1.1.2 | @uppy/status-bar | 3.3.1 | | @uppy/core | 3.10.0 | @uppy/thumbnail-generator | 3.1.0 | | @uppy/dashboard | 3.8.0 | @uppy/transloadit | 3.6.0 | | @uppy/drag-drop | 3.1.0 | @uppy/tus | 3.5.4 | | @uppy/drop-target | 2.0.5 | @uppy/unsplash | 3.3.0 | | @uppy/dropbox | 3.3.0 | @uppy/url | 3.6.0 | | @uppy/facebook | 3.3.0 | @uppy/utils | 5.7.5 | | @uppy/golden-retriever | 3.2.0 | @uppy/webcam | 3.4.0 | | @uppy/google-drive | 3.5.0 | @uppy/zoom | 2.3.0 | | @uppy/instagram | 3.3.0 | uppy | 3.24.0 | | @uppy/onedrive | 3.3.0 | | | - @uppy/box,@uppy/companion-client,@uppy/provider-views,@uppy/status-bar: fix type imports (Antoine du Hamel / #5038) - @uppy/aws-s3-multipart: mark `opts` as optional (Antoine du Hamel / #5039) - e2e: bump Cypress version (Antoine du Hamel / #5034) - @uppy/react: refactor to TS (Antoine du Hamel / #5012) - @uppy/core: refine type of private variables (Antoine du Hamel / #5028) - @uppy/dashboard: refine type of private variables (Antoine du Hamel / #5027) - @uppy/drag-drop: refine type of private variables (Antoine du Hamel / #5026) - @uppy/status-bar: refine type of private variables (Antoine du Hamel / #5025) - @uppy/remote-sources: migrate to TS (Merlijn Vos / #5020) - @uppy/dashboard: refine option types (Antoine du Hamel / #5022) - @uppy/dashboard: add new `autoOpen` option (Chris Grigg / #5001) - @uppy/core: fix some type errors (Antoine du Hamel / #5015) - @uppy/audio,@uppy/dashboard,@uppy/drop-target,@uppy/webcam: add missing exports (Antoine du Hamel / #5014) - meta: Bump webpack-dev-middleware from 5.3.3 to 5.3.4 (dependabot[bot] / #5013) - @uppy/dashboard: refactor to TypeScript (Antoine du Hamel / #4984) - @uppy/companion: improve error msg (Mikael Finstad / #5010) - @uppy/aws-s3-multipart: refactor to TS (Antoine du Hamel / #4902) - @uppy/dashboard: refactor to stable lifecycle method (Antoine du Hamel / #4999) - @uppy/companion: crash if trying to set path to / (Mikael Finstad / #5003) - @uppy/provider-views: fix `super.toggleCheckbox` bug (Mikael Finstad / #5004) - @uppy/aws-s3-multipart: fix escaping issue with client signed request (Hiroki Shimizu / #5006) - @uppy/drag-drop,@uppy/progress-bar: add missing exports (Antoine du Hamel / #5009) - @uppy/transloadit: migrate to TS (Merlijn Vos / #4987) - @uppy/utils: fix `RateLimitedQueue#wrapPromiseFunction` types (Antoine du Hamel / #5007) - @uppy/golden-retriever: migrate to TS (Merlijn Vos / #4989) - meta: Bump follow-redirects from 1.15.4 to 1.15.6 (dependabot[bot] / #5002) - meta: fix `resize-observer-polyfill` types (Antoine du Hamel / #4994) - @uppy/core: various type fixes (Antoine du Hamel / #4995) - @uppy/utils: fix `findAllDOMElements` type (Antoine du Hamel / #4997) - @uppy/status-bar: fix `recoveredState` type (Antoine du Hamel / #4996) - @uppy/utils: fix `AbortablePromise` type (Antoine du Hamel / #4988) - @uppy/core,@uppy/provider-views: Fix breadcrumbs (Evgenia Karunus / #4986) - @uppy/drag-drop: refactor to TypeScript (Antoine du Hamel / #4983) - @uppy/webcam: refactor to TypeScript (Antoine du Hamel / #4870) - @uppy/url: migrate to TS (Merlijn Vos / #4980) - @uppy/zoom: refactor to TypeScript (Murderlon / #4979) - @uppy/unsplash: refactor to TypeScript (Murderlon / #4979) - @uppy/onedrive: refactor to TypeScript (Murderlon / #4979) - @uppy/instagram: refactor to TypeScript (Murderlon / #4979) - @uppy/google-drive: refactor to TypeScript (Murderlon / #4979) - @uppy/facebook: refactor to TypeScript (Murderlon / #4979) - @uppy/dropbox: refactor to TypeScript (Murderlon / #4979) - @uppy/box: refactor to TypeScript (Murderlon / #4979) - @uppy/utils: migrate RateLimitedQueue to TS (Merlijn Vos / #4981) - @uppy/thumbnail-generator: migrate to TS (Merlijn Vos / #4978) - @uppy/screen-capture: migrate to TS (Merlijn Vos / #4965) - @uppy/companion-client: Replace Provider.initPlugin with composition (Merlijn Vos / #4977)
This fixes #5005 .
AWS SDK implementation
AWS SDK will escape all characters including "!'()*" such as not escaped by
encodeURI/encodeURIComponent
.This is aimed according to RFC 3986.
https://datatracker.ietf.org/doc/html/rfc3986
Those are escaped by AWS SDK's implementation, but it is complicated, so I investigated what code escapes that:
Escaping path
S3 presigner using command (for example, UploadPartCommand)
=> Ser(Request serializer) Middleware of UploadPartCommand
=> se_UploadPartCommand
=> resolvedPath
=> extended-uri-component
https://github.com/smithy-lang/smithy-typescript/blob/main/packages/smithy-client/src/extended-encode-uri-component.ts
Escaping query
@aws/signature-v4 (now deprecated, moved to @smithy/signature-v4)
SignatureV4
=> getCanonicalQuery
https://github.com/smithy-lang/smithy-typescript/blob/main/packages/signature-v4/src/getCanonicalQuery.ts
=> escapeUri
https://github.com/smithy-lang/smithy-typescript/blob/main/packages/util-uri-escape/src/escape-uri.ts
Uppy's current implementation
Escaping path
Using
encodeURI
to path directly=> All of "!'()*" chars are not escaped.
Escaping query
Using
URL#searchParams
to escape query=> Almost of special chars are escaped, but not only "*" is NOT.